Generating Hard Modal Problems for Modal Decision Procedures

Random generation of modal formulae is a viable method for generating problems for benchmarking modal decision procedures. However, previous work in this area has used a flawed generator that has resulted in questionable results. Fixing the generator changes the characteristics of the generated problems. The fixed generator can be used to generate hard problems that have more interesting modal properties than previous hard problem sets. Optimised decision procedures for propositional modal logics have recently been developed. These decision procedures are significantly faster than previous decision procedures for modal logics, to the extent that some problems that were impossible to solve using older decision procedures are trivial for the newer decision procedures. Further, due to the connection between propositional modal logics and description logics, there is a natural use for these optimised decision procedures in description logic classifiers. Now that such modal decision procedures are available, it is possible, and even necessary, to determine their actual performance, both relative and absolute, on hard, interesting modal problems. By hard problems, we mean problems that are hard, but not impossible, for one or, preferably, more current decision procedures, not necessarily problems that are theoretically hard. Several mechanisms have been proposed for generating such hard modal problems. These mechanisms generally break down into three groups: 1/ hand generation of classes of modal formulae, 2/ translation of actual problems into modal formulae, and 3/ random generation of hard modal formulae. The first group is best illustrated by the modal formulae used in the Tableaux’98 comparison of modal decision procedures [Heuerding and Schwendimann, 1996; Balsiger and Heuerding, 1998]. For this comparison, 54 parameterised formulae generators were created. Each generator took a size parameter and output a formula whose difficulty of proving (or equivalently determining the satisfiability of) was supposed to increase exponentially with the size of the parameter. The second group is illustrated by the translation of description logic knowledge bases, such as the GALEN knowledge base [Rector et al., 1994], or other knowledge descriptions, such as entity-relationship diagrams [Calvanese et al., 1998], into propositional modal logics. This mechanism was used in a comparison of description logic systems at DL’98 [Horrocks and Patel-Schneider, 1998a]. The third group is illustrated by the work of Giunchiglia and Sebastiani [Giunchiglia and Sebastiani, 1996a; 1996b] and Hustadt and Schmidt [Hustadt and Schmidt, 1997a;